Assessing the payback from health R & D: From ad hoc studies to regular monitoring

Chapter 1 : Introduction • The increasing demands for the benefits of payback from publicly funded R&D to be assessed are based partly on the need to justify or account for expenditure on R&D, and partly on the desire for information to assist resource allocation and the better management of R&D funds. The former consideration is particularly strong in relation to the R&D expenditure that comes out of the wider NHS budget. • In this report a range of categories of payback will be identified along with a variety of methods for assessing them. • The aim of the report is to make recommendations as to how the outcomes from health research might best be monitored on a regular basis. The specific context of the report is the NHS R&D Programme but many of the issues will be relevant for a wide range of funders of health R&D. • The introduction sets out not only a plan of the report but also suggests that readers familiar with the general arguments and existing literature may choose to jump to Chapter 6. Chapter 2 : Review of Existing Approaches to Assessing the Payback from Research • Existing work describes various approaches to valuing research. Some are ex ante and attempt to predict the outcomes of research being considered, others are ex post or retrospective. • The five categories of benefit or payback from health R&D that have been identified involve contributions: to knowledge; to research capacity and future research; to improved information for decision making; to the efficiency, efficacy and equity of health care services; and to the nation’s economic performance. These are shown in Table 1 of the report • The process by which R&D generates final outcomes can be modelled as a sequence. This includes primary outputs such as publications; secondary outputs in the form of policy or administrative decisions; and final outcomes which comprise the health and economic benefits. Feedback loops are also introduced and mitigate the limitations of a linear approach. • Qualitative and quantitative approaches can be used but there are immense problems with time lags and attributing outcomes, and sometimes even outputs, to specific items of research funding. • Four common methods of measuring payback can be used. Expert review, by peers or, sometimes, users is the traditional way of assessing the quality of research. Bibliometric techniques can involve not only counting publications but also using datasets such as the Science Citation Index and Wellcome’s Research Outputs Database (ROD). The various methods of economic analysis of payback are difficult to undertake given the costs and problems of acquiring relevant information and estimating benefits. Social science methods include case studies, which can provide useful information but are resource intensive, and questionnaires to researchers and potential research users. Chapter 3 : Characteristics of a Routine Monitoring System • In moving from ad hoc or research studies of payback towards a more regular monitoring it is noted that whereas there has always been a tradition of evaluation of research, in the public services in general there is now a greater emphasis on audit and performance measurement and indicators. A review of these various systems suggests we should be looking to develop a system of outcomes monitoring that incorporates performance indicators (PIs) and measurement rather than an audit system that is trying to monitor activities against predetermined targets. • Standard characteristics of performance measurement systems do not necessarily apply to research where, for example, there are non-standard outputs. Difficulties have arisen in the USA in attempting to apply the Government Performance and Results Act to research funding agencies. It is shown that because the findings of basic research, in particular, enter a knowledge pool in which people and ideas interact, it is difficult to use a PIs’ approach to track eventual outcomes. However, for some types of health research it has proved more feasible to trace the flow between research outputs and outcomes. • An outcomes monitoring system could be useful if it met the following criteria: relevant to, with as comprehensive coverage as possible of, the funders objectives; relevant to the funder’s decision making processes; encourages accurate compliance; minimises unintended consequences; and has acceptable costs. Chapter 4 : Differences Between Research Types • The range of differences between types of research can be relevant for the design of a routine monitoring system. The OECD distinguishes between basic research, applied research and experimental development. Most DH/NHS research is applied. There might be more of a tradition of publication of findings in applied research in health than in other fields. Nevertheless, the publication and incentives patterns operating in basic research mean that it would be inappropriate to use bibliometric indicators in a simple way across all fields even in health research. • Despite having some differences from health research in publication patterns and in the detailed categories of payback, the broad approach proposed in Chapter 6 could be applied to social care research. • Research that is commissioned, especially by the government, has some of the minimum conditions built into it that are associated with outcomes being generated, in particular because the funder has identified that a contribution in this area will be valuable. Chapter 5 : What Units of Research? • The term programme has various meanings including being used to describe a collection of projects on a common theme and to describe a block of funding for a research unit. • Three main streams or modes of funding can be identified: projects, which are administratively grouped into programmes including a responsive programme; institutions/centres/units; individual researchers. These 3 streams are displayed in Figure 1. It is probable that the regular data-gathering for a monitoring system would operate at the basic level of each stream or mode. • Previous work demonstrates that the full range of benefits can sometimes be applied at the level of projects, either in the responsive mode or in programmes, through the use of questionnaires to researchers. Expert and user review and user surveys have also been applied. • Institutions and centres increasingly have experience not only of traditional periodic expert review but also of producing annual reports, although there are debates about what dimensions to include in such reviews and reports. • Individuals in receipt of research development awards have completed questionnaires during and after the awards. These concentrate on the development of research capacity but can go wider. Chapter 6 : A Possible Comprehensive Outcomes Monitoring System • The proposed system is intended for DH/NHS to monitor the outcomes from its R&D in order to justify the R&D expenditure and assist with managing the portfolio. More detailed information is required for the latter purpose. • We propose a multidimensional approach be adopted to cover all the dimensions of payback and that information be gathered from three sets of sources and Table 3 shows which methods would cover which output/outcome categories. • Firstly, possibly annually, a questionnaire (possibly electronic) covering most payback categories should gather data from the basic level of each funding stream ie. from lead researchers of projects, from research institutions/centres, and from individual award holders. • Secondly, supplementary information should be gathered from external databases (including the citation indices and Wellcome’s ROD). • Thirdly, a range of approaches ie. user surveys, reviews by experts and peers, case studies including economic evaluations, and analysis of sources used in policy documents such as NICE guidelines, would be undertaken on a sample basis. They would provide not only supplementary information but, as with the external databases, would also verify the data collected directly from researchers. • These proposals can be evaluated against the criteria set out in Chapter 3: • The system is relevant to DH’s objectives of generating payback in a range of categories. • Various problems have to be overcome before the system could be fully decision relevant. Firstly it might be necessary to ask researchers to apportion the contribution made to specific outputs from various funding streams. Second, to be decision relevant the information would have to be analysed and presented in a manner consistent with funders’ decision making processes. This would involve a) showing how for each outcome and output, for example publications, data from one project or stream could be compared with those from another and b) demonstrating how different outputs and outcomes could be aggregated. • The questions of accuracy of data, minimisation of unintended consequences and the acceptability of the net costs are also addressed. Chapter 7 : Research and Monitoring • Whilst this report is primarily concerned with moving from ad hoc studies towards a routine monitoring system there are issues that need further research. • Before embarking on full implementation the feasibility needs to be tested of items such as on-line recording of data and asking researchers to attribute proportions of research outputs to separate funding agencies. • Once the system is implemented the value of some items can be better assessed, for example the additional value provided by self reporting of publications beyond that gained from relying on external databases. • The data provided by the system would provide opportunities for further payback research on, for example, links between publications and other categories of payback. • Some items such as network analysis could potentially be added to the monitoring system after further examination of them. • Finally the benefit from the monitoring system itself should be assessed.Department of Health; Wellcome Trus

Evaluation of the NHS R & D implementation methods programme

Chapter 1: Background and introduction • Concern with research implementation was a major factor behind the creation of the NHS R&D Programme in 1991. In 1994 an Advisory Group was established to identify research priorities in this field. The Implementation Methods Programme (IMP) flowed from this and its Commissioning Group funded 36 projects. Funding for the IMP was capped before the second round of commissioning. The Commissioning Group was disbanded and eventually responsibility for the programme passed to the National Co-ordinating Centre for NHS Service Delivery and Organisation R&D (NCCSDO) which, when most projects had finished, asked the Health Economics Research Group (HERG) to conduct an evaluation. This was intended to cover: the quality of outputs; lessons to be learnt about the communication strategy and the commissioning process; and the benefits or payback from the projects. As agreed, the evaluation also addresses the questions of whether there should be a synthesis of the findings from the IMP and any further assessment of payback. Chapter 2: Methods • We adopted a wide range of quantitative and qualitative methods in the evaluation. They included: documentary analysis; interviews with key actors; questionnaires to the funded lead researchers; questionnaires to potential users; and desk analysis. Chapter 3: The outputs from the programme • As in previous assessments of research programmes, we first examined the outputs in terms of knowledge production and various items related to capacity to conduct further research. Although there was a high response rate to the questionnaire to lead researchers (30/36), missing responses mean that the data given below are incomplete. In the case of publications, however, we also made some use of data previously gathered by the programme office. • We attempted to identify publications that were in some way a specific product of IMP funding. About half (59) of the publications from the IMP projects are articles in peer reviewed journals. The journal used most frequently for publication, the BMJ, is also the one with the highest journal impact factor score of those publishing articles specifically from the programme. The recent publication datesof many articles reduces the value of citation analysis. Nevertheless, one article, Coulter et al, 1999, has already been cited on 53 occasions. Important publications, including No Magic Bullets (Oxman et al, 1995), are also associated with preliminary work undertaken for the IMP to assist priority setting. • Fifteen projects, with grants of over £1.3 million, have been awarded to IMP researchers by other funders for follow-on studies connected in some way to the IMP. We also collected details about some non-IMP researchers who are building on the IMP projects. • Research training provided in at least nine of the funded IMP projects is associated with higher/research degrees, including three MDs and four PhDs, that have been awarded or are being completed. Chapter 4: Disseminating and using the research findings • Limited thought had been given by the Implementation Methods Programme to dissemination strategies, but many of the individual researchers were active here. In response to the questionnaires, lead researchers reported making 92 presentations to academic audiences and 104 to practitioner/service groups. Some lead researchers showed that their effective dissemination led to utilisation of the findings. • The Commissioning Group gave some thought to the likely use that could be made of individual research projects, but there was limited systematic analysis of how the findings as a whole would be taken forward. Achieving impact is difficult in this complex field and less than a third of lead researchers claimed to have done so, but about half thought impact could be expected. Based mainly on reports from lead researchers, we give a brief account of how the findings from six projects are being utilised. • We sent electronic questionnaires to groups of potential users of selected projects but this produced a very low response rate. Our postal survey to Heads of Midwifery/researchers in perinatal care produced a higher response of 44%. Amongst those who did respond, there is quite a high level of knowledge about some of the programme’s projects and some level of existing and potential utilisation. We suggest, however, that in some cases there are difficulties in identifying how far the respondent’s focus is on the findings from the original research projects, and it how far it is on the impact of the IMP study that is about ways of influencing the uptake of such findings. Comments from several respondents showed strong support for the cutting edge nature of some of the research. Others, however, also indicated why findings might not be utilised by some practitioners. Several respondents advocated greater dissemination of the IMP. Chapter 5: Comparing applications with outputs • We attempted to compare the scores given to project applications with those given to projects based on their outputs. This exercise faced various problems. The final reports from all completed projects had in theory already been reviewed and given scores for their quality and relevance. In practice, not all final reports received scores. We added a refinement by giving further scores that incorporated the additional information we gathered about both publications and any uptake of the research findings. • Various limitations meant that we conducted this analysis on just 19 of the 36 projects. Nevertheless, the wide range of scores given to the outputs from projects indicates that some were much more successful than others. Our rather limited evidence suggests that there is some correlation between the scores for applications and those for outputs but it is small, which could be related to the difficulties encountered during commissioning. Chapter 6: Lessons learnt about the commissioning process • Those who established the IMP were aware that it was a different type of research field from those previously addressed within the NHS R&D Programme, but one regarded nonetheless as important. Within the NHS R&D Programme at that time a standard clinical RCT approach was strongly favoured. There was also, as ever, a need for quick results. • In developing an understanding of implementation the Advisory Group (AG) conducted cutting edge analysis, consulted widely and drew on a wide range of disciplines. Our interviewees generally took the view that the AG worked well in setting priorities and went as far as it could at the time, especially given the time constraints. • Based on our field work and analysis we identified a series of lessons that might inform future exercises. More attention was required to ensure that all relevant background disciplines were adequately taken into account in setting priorities and commissioning research. Some of these processes needed to be given more time than was available. Consultation needed to be organised in a sufficiently selective way to be of maximum benefit in such a complex area. A time-limited programme was not the most appropriate way to cover a field such as this. • In relation to the commissioning of the projects, we identified issues about the composition of commissioning groups and how people from different backgrounds (researchers, practitioners, managers and patient representatives) should best be involved. • In this new field the Commissioning Group (CG) had to work closely with applicants to develop some of the research applications. This raised issues about how, and when, this process should be handled. • Despite its own rationale, and for a variety of reasons, including the disbanding of the CG, the Implementation Methods Programme never developed an implementation or communication strategy for its own findings. • The general conclusion of those who had been involved with the IMP was that it worked as well as it could at the time, and that various important projects were commissioned. But it was only a start. Chapter 7: Should a synthesis of the findings from the programme and further payback analysis be undertaken? • From interviews and questionnaires we identified widespread, but not total, agreement that there should be some type of synthesis of the findings from the IMP. There is more debate about the form such a synthesis should take. There is some support for a more limited type of stock taking, but also wider backing for the inclusion of many different elements. These could include: a conceptual map of the field of research implementation; an exploration of how the findings from the IMP fit into the context of research implementation today; and an assessment of how far work is still needed in those areas where no projects were funded. One possible suggestion that might incorporate much of this thinking is for the establishment of a group or commission of leading researchers in the field. Their investigation could incorporate all these elements and attempt to show how the issues could be advanced. • We suggest that further work on assessing the payback from the IMP is probably not worthwhile unless it is undertaken as part of a wide-ranging synthesis. Chapter 8: Conclusions, lessons and recommendations • We conclude that the IMP was seen by many of those involved as a new and exciting field. Looking back, they were generally positive about what was started through the IMP. It commissioned a series of projects that produced some important, rigorous, and cutting edge research, at least some of which is making an impact. But this is a complex area in which traditional clinical research, health services research and the social sciences all have a role to play. A unique set of difficulties, as well as opportunities, was faced by those responsible for taking the programme forward. The intellectual challenges of constructing a programme to cover such a vast area with diverse and sometimes conflicting conceptual and methodological perspectives, were compounded by practical problems. These included the capping of the programme’s funding and the premature winding up of the Commissioning Group. As a result, this complex programme, which arguably needed better support than its more clinically orientated predecessors, did not receive it at some stages. Those involved in the programme had a considerable task – the difficulties of which were not completely appreciated at the time. They are clearer in retrospect and feed into the lessons and recommendations presented here, but it is recognised that a programme such as the SDO is already adopting some of the steps. • In relation to research commissioning and communication strategies for research programmes in general, we suggest it could be helpful if protocols were drawn up to cover certain potential difficulties. These include the remit and role of the various stakeholders represented on commissioning groups and the extent to which commissioning groups should be expected to support applicants with their proposals. Perhaps the key general lesson from this evaluation is the need for research programmes to have a proper communication strategy. This should target dissemination at relevant audiences and stress the desirability for contact to be made with potential users as early as possible in the process of devising a project. • Our other recommendations are more specifically relevant when the SDO Programme is considering an area such as implementation methods research. It would be desirable for more time to be made available for preparatory work than was allowed for the IMP and also scope provided for the programme to be able to re-visit issues and learn from early results. It is difficult to incorporate all the analysis that is required if a programme is operating in a time-limited way. • Our conclusion that research implementation is a crucial area for the NHS R&D Programme leads to the recommendation that more R&D activity is needed in this field in order to assist delivery of some key NHS agenda items. As a preliminary step, there is certainly scope for a type of stock taking of the findings from the IMP. On balance there seems also to be an argument for conducting a synthesis of work in the implementation field that goes beyond a mere collation of findings from the specific projects funded. If undertaken, it should fundamentally examine the current NHS needs for research on implementation and how they could be addressed in the light of the findings from the IMP and elsewhere. • Finally, we recommend that more attention should be given to the timing of evaluations such as this and that a phased approach should be adopted. Furthermore, researchers should be informed at the outset of their project about the likely requirements that might be placed upon them in terms of responding to requests for information by those conducting an evaluation.National Co-ordinating Centre for NHS Service Delivery and Organisation R&D (NCCSDO

A novel method of combining blood oxygenation and blood flow sensitive magnetic resonance imaging techniques to measure the cerebral blood flow and oxygen metabolism responses to an unknown neural stimulus.

Simultaneous implementation of magnetic resonance imaging methods for Arterial Spin Labeling (ASL) and Blood Oxygenation Level Dependent (BOLD) imaging makes it possible to quantitatively measure the changes in cerebral blood flow (CBF) and cerebral oxygen metabolism (CMRO(2)) that occur in response to neural stimuli. To date, however, the range of neural stimuli amenable to quantitative analysis is limited to those that may be presented in a simple block or event related design such that measurements may be repeated and averaged to improve precision. Here we examined the feasibility of using the relationship between cerebral blood flow and the BOLD signal to improve dynamic estimates of blood flow fluctuations as well as to estimate metabolic-hemodynamic coupling under conditions where a stimulus pattern is unknown. We found that by combining the information contained in simultaneously acquired BOLD and ASL signals through a method we term BOLD Constrained Perfusion (BCP) estimation, we could significantly improve the precision of our estimates of the hemodynamic response to a visual stimulus and, under the conditions of a calibrated BOLD experiment, accurately determine the ratio of the oxygen metabolic response to the hemodynamic response. Importantly we were able to accomplish this without utilizing a priori knowledge of the temporal nature of the neural stimulus, suggesting that BOLD Constrained Perfusion estimation may make it feasible to quantitatively study the cerebral metabolic and hemodynamic responses to more natural stimuli that cannot be easily repeated or averaged

Interpreting Oxygenation-Based Neuroimaging Signals: The Importance and the Challenge of Understanding Brain Oxygen Metabolism

Functional magnetic resonance imaging is widely used to map patterns of brain activation based on blood oxygenation level dependent (BOLD) signal changes associated with changes in neural activity. However, because oxygenation changes depend on the relative changes in cerebral blood flow (CBF) and cerebral metabolic rate of oxygen (CMRO2), a quantitative interpretation of BOLD signals, and also other functional neuroimaging signals related to blood or tissue oxygenation, is fundamentally limited until we better understand brain oxygen metabolism and how it is related to blood flow. However, the positive side of the complexity of oxygenation signals is that when combined with dynamic CBF measurements they potentially provide the best tool currently available for investigating the dynamics of CMRO2. This review focuses on the problem of interpreting oxygenation-based signals, the challenges involved in measuring CMRO2 in general, and what is needed to put oxygenation-based estimates of CMRO2 on a firm foundation. The importance of developing a solid theoretical framework is emphasized, both as an essential tool for analyzing oxygenation-based multimodal measurements, and also potentially as a way to better understand the physiological phenomena themselves. The existing data, integrated within a simple theoretical framework of O2 transport, suggests the hypothesis that an important functional role of the mismatch of CBF and CMRO2 changes with neural activation is to prevent a fall of tissue pO2. Future directions for better understanding brain oxygen metabolism are discussed

Economic evaluation of ASCOT-BPLA: Antihypertensive treatment with an amlodipine-based regimen is cost-effective compared to an atenolol-based regimen

Copyright © 2010 BMJ Publishing Group Ltd & British Cardiovascular Society. Internal or personal use of this material is permitted. However, permission to reprint/republish this material must be obtained from the Publisher.Objective: To compare the cost effectiveness of an amlodipine-based strategy and an atenolol-based strategy in the treatment of hypertension in the UK and Sweden. Design: A prospective, randomised trial complemented with a Markov model to assess long-term costs and health effects. Setting: Primary care. Patients: Patients with moderate hypertension and three or more additional risk factors. Interventions: Amlodipine 5–10 mg with perindopril 4–8 mg added as needed or atenolol 50–100 mg with bendroflumethiazide 1.25–2.5 mg and potassium added as needed Main outcome measures: Cost per cardiovascular event and procedure avoided, and cost per quality-adjusted life-year gained. Results: In the UK, the cost to avoid one cardiovascular event or procedure would be €18 965, and the cost to gain one quality-adjusted life-year would be €21 875. The corresponding figures for Sweden were €13 210 and €16 856. Conclusions: Compared with the thresholds applied by NICE and in the Swedish National Board of Health and Welfare’s Guidelines for Cardiac Care, an amlodipine-based regimen is cost effective for the treatment of hypertension compared with an atenolol-based regimen in the population studied.The study was supported by the principal funding source, Pfizer, New York, USA

Discovery of a supernova associated with GRB 031203: SMARTS Optical-Infrared Lightcurves from 0.2 to 92 days

Optical and infrared monitoring of the afterglow site of gamma-ray burst (GRB) 031203 has revealed a brightening source embedded in the host galaxy, which we attribute to the presence of a supernova (SN) related to the GRB ("SN 031203"). We present details of the discovery and evolution of SN 031203 from 0.2 to 92 days after the GRB, derived from SMARTS consortium photometry in I and J bands. A template type Ic lightcurve, constructed from SN 1998bw photometry, is consistent with the peak brightness of SN 031203 although the lightcurves are not identical. Differential astrometry reveals that the SN, and hence the GRB, occurred less than 300 h_71^-1 pc (3-sigma) from the apparent galaxy center. The peak of the supernova is brighter than the optical afterglow suggesting that this source is intermediate between a strong GRB and a supernova.Comment: 11 pages, 3 figures, submitted to ApJ Letter

Straight Spouses Speak Out: Implications for Gay and Lesbian Marriage

This article highlights the experiences of straight spouses who have weathered the ìcoming outî storm of their partners and their views of same-gender marriage. Experiences of shock, denial, and for some, relief, are chronicled, along with post-disclosure stories of coping and family life outcomes. An overview of mixed-orientation marriage and children from these marriages are discussed

Understanding the fidelity effect when evaluating games with children

There have been a number of studies that have compared evaluation results from prototypes of different fidelities but very few of these are with children. This paper reports a comparative study of three prototypes ranging from low fidelity to high fidelity within the context of mobile games, using a between subject design with 37 participants aged 7 to 9. The children played a matching game on either an iPad, a paper prototype using screen shots of the actual game or a sketched version. Observational data was captured to establish the usability problems, and two tools from the Fun Toolkit were used to measure user experience. The results showed that there was little difference for user experience between the three prototypes and very few usability problems were unique to a specific prototype. The contribution of this paper is that children using low-fidelity prototypes can effectively evaluate games of this genre and style

Spatial audio in small display screen devices

Our work addresses the problem of (visual) clutter in mobile device interfaces. The solution we propose involves the translation of technique-from the graphical to the audio domain-for expliting space in information representation. This article presents an illustrative example in the form of a spatialisedaudio progress bar. In usability tests, participants performed background monitoring tasks significantly more accurately using this spatialised audio (a compared with a conventional visual) progress bar. Moreover, their performance in a simultaneously running, visually demanding foreground task was significantly improved in the eye-free monitoring condition. These results have important implications for the design of multi-tasking interfaces for mobile devices